Autograd



In [6]:

    
import time

Have to install autograd module first: pip install autograd



In [9]:

    
import autograd.numpy as np   # Thinly-wrapped version of Numpy
from autograd import grad

EX1, Normal Numpy



In [21]:

    
def tanh(x):
    y = np.exp(-x)
    return (1.0 - y) / (1.0 + y)

start = time.time()

grad_tanh = grad(tanh)
print ("Gradient at x = 1.0\n", grad_tanh(1.0))

end = time.time()
print("Operation time:\n", end-start)









    



Gradient at x = 1.0
 0.393223866483
Operation time:
 0.0029020309448242188

EX2-1, Taylor approximation to sine function



In [22]:

    
def taylor_sine(x):  
    ans = currterm = x
    i = 0
    while np.abs(currterm) > 0.001:
        currterm = -currterm * x**2 / ((2 * i + 3) * (2 * i + 2))
        ans = ans + currterm
        i += 1
    return ans

start = time.time()

grad_sine = grad(taylor_sine)
print ("Gradient of sin(pi):\n", grad_sine(np.pi))

end = time.time()
print("Operation time:\n", end-start)









    



Gradient of sin(pi):
 -0.9998995297042174
Operation time:
 0.0029990673065185547

EX2-2, Second-order gradient



In [23]:

    
start = time.time()

#second-order
ggrad_sine = grad(grad_sine)
print ("Gradient of second-order:\n", ggrad_sine(np.pi))

end = time.time()
print("Operation time:\n", end-start)









    



Gradient of second-order:
 0.0004451602382100628
Operation time:
 0.0070531368255615234

EX3, Logistic Regression

A common use case for automatic differentiation is to train a probabilistic model.
A Simple (but complete) example of specifying and training a logistic regression model for binary classification:



In [24]:

    
def sigmoid(x):
    return 0.5*(np.tanh(x) + 1)

def logistic_predictions(weights, inputs):
    # Outputs probability of a label being true according to logistic model.
    return sigmoid(np.dot(inputs, weights))

def training_loss(weights):
    # Training loss is the negative log-likelihood of the training labels.
    preds = logistic_predictions(weights, inputs)
    label_probabilities = preds * targets + (1 - preds) * (1 - targets)
    return -np.sum(np.log(label_probabilities))

# Build a toy dataset.
inputs = np.array([[0.52, 1.12,  0.77],
                   [0.88, -1.08, 0.15],
                   [0.52, 0.06, -1.30],
                   [0.74, -2.49, 1.39]])
targets = np.array([True, True, False, True])

# Define a function that returns gradients of training loss using autograd.
training_gradient_fun = grad(training_loss)

# Optimize weights using gradient descent.
weights = np.array([0.0, 0.0, 0.0])
print ("Initial loss:", training_loss(weights))

for i in range(100):
    weights -= training_gradient_fun(weights) * 0.01

print  ("Trained loss:", training_loss(weights))









    



Initial loss: 2.77258872224
Trained loss: 0.389007543156



In [ ]: